Contents

2. SLS Lecture 2 : Shell Part II

https://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Shell_Island_1985.jpg/1920px-Shell_Island_1985.jpg

Getting our hands dirty.

As powerful as UNIX and the shell are they are also old and dirty... have had plenty of time to accumulate some Barnacles.

Our goals:

  1. Basic proficiencies in

  • using the shell as a programming language

    • core to automating your UNIX life

      • interactively : one liners

      • scripts : small and large programs

  • navigating and working with the file system

  • running and managing running programs

    • IO redirection and pipes

    • processes

  • understand basics of UNIX credentials

    • user and groups names and ids, file permissions, and who “root” is

  1. Enable you to learn more

  • know enough so that you can experiment

  • know where you can look for more info

  • know enough so you can appreciate and learn from others

    • idioms, tricks, and hacks

    • scripts

  1. Understand how things work so that you have the context for

    • working efficiently with large bodies of ascii files

      • at the command line

      • and in editors like vim and emacs

    • make

    • git

2.1. The Shell as a Programming Language

** The Glue that holds the UNIX Universe together **

Basic Syntax : Notes

  • comments

  • echo and printf

  • simple variable syntax and usage

    • quoting to control splitting and expansion

    • many more fancy variable expansions and special variables : help variables but plenty more

  • basic programming constructs

    • for lists of words

    • commands and exit codes

      • (())

      • [[]]

      • [ ]

  • interactive vs scripts

    • history and arrow keys

    • scripts

      • ascii file with lines of shell commands

      • mark as executable and put in path or name explicitly

3. Shell Programming

# Comments
# a word that starts with # and everthing else till end end of the line is ignored

# echo - basic built in for outputting
echo hello

# shell variables and variable expansion has lots of subtly but lets start simple
i=hello

# remember shell expansions will happen to the line prior the "echo" built in being
# executed

echo $i  # $ -- triggers parameter/variable expansion

# quoting used to control expansion and splitting

x=goodbye and farewell

# single quotes supresses all expansion and splitting - literal value of each character
# single quotes cannot contain single quotes
x='goodbye and farewell'
echo $x

x='$i goodbye and farewell'
echo $x

# double quotes partially suppress expansions but allow some like $
x="$i goodbye and farewell"

# ; can be used to join commands as a list on the same line
x="$i goodbye and farewell"; echo $x

# ok we will look at more variable related syntax and expansions later
# lets try out some loops: for, until, while, break

# for name in word ; do commands; done
for color in red green black brown; do  echo "color=$color"; done

# or
for color in read green black brown
do
    echo "color=$color"
done


# the shell also has support for integers and arithmetic
for ((i=0; i<5; i++)); do echo "i=$i"; done

# actually (()) tells the shell to do arithmetic in general
(( j++ )); echo $j

# shell arithemetic supports comparisons as well
(( j<0 ))

# special parameter ? is exit code of last command/pipe
echo $?

(( j>0 )); echo $?

# shell convention is exit value of 0 is success and non-zero if failure
# becare oposite to what you might be used to 

if (( j == 0 )); then echo "j is zero"; elif ((j<10)); then echo "j is under 10"; else "j is something else $j"; fi

# beyond arithmetic expressions bash support a more general [[]] expression
# return code will still indicate success vs failure bue allow non-integer logic
[[ $i == "hello" ]]; echo $?

# special case of logical and and or to conditional commbine commands 
(( j<0 )) && echo "j is < 0"
(( j<0 )) || echo "j is >= 0"
# could have written using if

# conditions are  just tests of exit code of command eg.  success versus failure
# So any command (internal or external) written to correctly indicate success and failure with exit code
# can be used as condition

# shell also has case and select statements

4. File and Directories

# Current working directory
pwd

# What is in the current directory?
#  Another expansion - Filename expansion the shell does for us.
echo *
echo /*


# ls is a powerfull external command for list contents of directories
ls
man ls
ls -l
ls -lrt
ls -lhrt
ls -1

# change working directory
#
cd /
pwd
ls
cd home
pwd
ls
cd jovyan
cd /home/jovyan
pwd
cd /usr
pwd
ls -l /var/*
pwd
cd

# . and ..
ls .
ls ..

# full path
ls /bin/ls
ls /var/log

# combining
cd /var/log/../g*

# short cut to name home
# ~ expansion
echo ~
ls ~
echo ~*

# . prefix name to hide
ls -a



# creating directoreies
# man mkdir
mkdir ~/bin
mkdir $HOME/tmp
cd ~
mkdir Classes
mkdir Classes/CS
mkdir Classes/CS/400
MYDIR=Classes/CS/400
for ((i=0; i<10; i++)); do echo mkdir $MYDIR/$i; done

# create an empty file
# man touch
touch foo
ls
rm foo
#for ((i=0; i<10; i++)); do echo touch $MYDIR/$i; done
for d in $(ls $MYDIR); do rm  $MYDIR/$d/README.md; done
rm -rf Classes

# find is also a powerfull command

4.1. Running and Managing Programs (Commands)

4.1.1. Processes, Files and Streams

  • What is a process in more detail and what can it do

    • a running program

      • process can launch another processes

        • bash is implemented to start commands as process (when needed)

        • passes command line arguments and environment variables

    • Processes, Files and channel/stream model of I/O

      • files and Kernel objects

        • read, write

        • everything is a file

      • Streams: Processes and Files

        • open : attach a file as a stream

        • file descriptors/handles : stream

          • read and write bytes to a stream

          • close

          • dup

        • standard input, standard output, standard error

      • Shell Syntax:

        • echo 'Hello world' > hello

        • cat hello

        • cat < ./hello

    • We now can understand what a pipe is

      • pipe file object

      • Process 1 stdout into pipe and Process 2 stdin from pipe

      • Shell Syntax:

        • ‘ls -1 | wc -l’

        • ‘ls -1 | grep ‘^l*’ | wc -l’

        • mknod mypipe p and mkfifo mypipe

4.2. Credentials and file permissions

  • Process have id’s associated with them

    • a single user id : id

      • a single number that maps to a string user name (/etc/passwd)

    • set of group ids

      • user has a primary group but can be in many secondary groups

      • each has a number that maps to a name (/etc/group)

      • each group can have many users

    • ps auxgww

      • process’s inherit their ids from their parent

  • Files have id’s and permissions ls -l, chmod, chown, and chgrp

    • user, group, other -> read, write, execute

    • kernel ensures that process id’s and requested operations match permissions

  • Process management

    • ps - look at all processes

    • The shell and its children

      • & : foreground and background

      • ctrl-z

      • jobs

    • so we know how to start process, list them how about stopping

    • kill

      • signals

      • ctrl-c

      • or without prejudice

4.3. Variables in detail

  • has a name and stores a value

  • variables are assigned a value with name=[value]

    • Note cannot have spaces between either name = or = and value

    • by default variable are “string” variables

    • NULL/empty string is a valid value eg. x=

    • value of an assignment is subject to the following expansions:

      • tilde

      • parameter and variable expansion

      • quote removal

      • if variable has integer attribute then arithmetic expansion is done

    • one more attributes that can be assigned with the declare builtin

    • no need to declare

      • eg. assignment to a non-exiting variable creates it

    • can mark some variables for export (environment variables)

      • “exported” variables are copied via UNIX environment to child processes

        • Environment a set of strings of the form : “NAME=STRING”

        • Bash has the kernel copy the current set of exported variables when it launches a new program. Programs can inspect these values and customize their behavior. (eg. locale)

        • export <Variable>[=Value] (can do the same with declare -x)

  • variable expansion and quotes - echo $x   $y vs echo 'echo $x   $y' vs echo “\(x \)y”      -echo “$x $y”`

  • some special unnamed variables (called parameters)

    • actually variables are technically parameters with names

    • 1 - n – positional parameters expand to ordered arguments of script or function

    • @ and * – expands to positional parameters however with differing expansion rules

    • # – expands to number of positional parameters

    • ? – expands to exit status of last command/pipeline

    • others – -, $, !, 0

  • Advanced: declare

    • integers

      • declare -i myvar; myvar=myvar+1; echo $myvar vs not using the declare -i?

    • indexed arrays

    • associative arrays

    • ref variables

    • other built-ins that are special forms of declare

      • typeset (same as declare)

      • export (like using declare -x)

      • readonly (like using declare -r)

      • +=

      • unset

  • things that look like variables but are really fancy expansions

    • $ actually triggers three types of expansion

      1. parameter expansion

      2. command substitution

      3. arithmetic expansion

    • Parameter expansion

      • simple: $name or more explicitly ${name}

      • many advanced forms: Here are a few that I constantly use

        • setting default value eg. foo=${foo:-START}

        • delete prefix eg. foo=ABCA.txt; echo ${foo##*.}

        • delete suffix eg. foo=ABCA.txt; echo ${foo%%.txt}

        • substitution/deletion eg. foo=ABCA.txt; echo ${foo//A/O} or echo ${foo//A/}

    • Command substitution

      • very useful run a command and treat its output as a value

      • foo=$(ls); echo $ls or n=$(ls -1 | wc -l); echo $n

    • Arithmetic expansion: $(()), or (()) or let, or if integer types variables

      • Shell knows how to do basic integer math and some binary/bit operators

        • ++, --, +, -, !, ~, **, *, /, %, <<, >>

        • <=, >=, <, >, ==, !=

        • &, ^, |, &&, ||

        • expr ? expr : expr

        • =, *=, /=, %=, +=, -=, <<=, >>=, &=, ^=, |=

        • number formats:

          • prefix: 0 - octal, 0x - hex, [#base]number – eg. ((x=2#101)); echo $x

See. https://www.gnu.org/software/bash/manual/bash.html#Shell-Commands 2. Breakdown complex commands into simple commands

  • complex commands are a composition of simple commands:

    • pipelines

      • connecting inputs and outputs of commands together using |

      • ls -1 /bin | wc -l

    • lists

      • one or more pipelines separated by one of: ;, &&, || or & and terminated by ; or ; or & or a newline.

        • echo "about to run ls"; ls; echo "done running ls"

        • cmd1 && cmd2 : run cmd2 if cmd1 returns “success”

        • cmd1 || cmd2 : run cmd2 if cmd1 returns “not success” aka “failure”

        • cmd1 & cmd2 : run cmd1 asynchronously in the background and run cmd2 “normally”

          • lots more to say about asynchronous commands

    • compound commands: Shell programming constructs (these can span lines)

      • loops: until, while, for, do, done, break, continue

        • there are two type of for loops

      • conditionals: if, then, elif, else, fi, case, select, (()), [[]]

      • grouping: (), {}

  • functions

../_images/dirtree.png

/ \(+\) home \(+\) jonathan \(+\) Classes \(+\) CS \(+\) 210 \(+\) Assignment 1 \(+\) Problem 1

/ \(+\) home \(+\) jonathan \(+\) Classes \(+\) CS \(+\) 210 \(+\) Assignment 2 \(+\) Problem 1

/ \(+\) home \(+\) jonathan \(+\) Classes \(+\) CS \(+\) 210 \(+\) Assignment 3 \(+\) Problem 1

“Full” path name: join independent components with the “/” character. So the above three files as proper unix path names would be: